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Preface 


The  Department  of  Defense’s  High  Performance  Computing  Modernization  Program  (HPCMP) 
develops  supercomputing  resources  and  expertise  to  support  the  Department’s  research, 
development,  test  and  evaluation  (RDT&E).  Originally  managed  by  the  Office  of  Secretary  of 
Defense  (OSD),  the  HPCMP  was  transferred  from  OSD  to  the  Army  in  fiscal  year  (FY)  2012. 
This  transfer  was  accompanied  by  a  reduction  in  funds:  In  FY12,  the  program  was  provided  with 
$183  million,  which  represented  roughly  a  $40  million  reduction  when  compared  with  the 
previous  fiscal  year.  Since  then,  Congress  added  funding  to  the  HPCMP  to  ensure  continuity 
with  pre-FY12  funding  levels. 

In  fall  2014,  Congress  asked  the  Department  of  Defense  to  begin  thinking  about  a  longer- 
term  solution  to  this  problem.  In  response  to  this  request,  the  Army  submitted  a  report  that 
offered  several  courses  of  action  for  closing  the  budget  gap,  which  ranged  from  eliminating  two 
of  the  program’s  five  centers  to  reducing  research  investments  in  network  and  software 
development. 

To  help  prepare  for  this  decision,  the  Air  Force  Research  Faboratory  (AFRF),  which 
manages  two  of  the  five  computing  sites,  asked  RAND  to  perform  research  on  one  of  these  two 
sites,  the  Maui  High  Performance  Computing  Center  (MHPCC).  Specifically,  AFRF  asked 
RAND  to  assess  the  computational  requirements  of  Pacific  area  of  responsibility  government 
entities,  notably  U.S.  Pacific  Command,  with  respect  to  supercomputing  assets  and  capabilities. 
The  findings  and  recommendations  contained  within  this  report  are  designed  to  provide  AFRF 
with  important  context  about  potential  future  missions  and  use  cases  for  MHPCC  as  Congress, 
the  Army,  and  AFRF  decisionmakers  work  toward  a  budget  gap  resolution.  The  research  does 
not  attempt  to  pass  judgment  as  to  how  well  MHPCC  supports  the  existing  HPCMP  RDT&E 
mission,  nor  does  it  offer  a  detailed  appraisal  of  the  greater  HPCMP ’s  mission,  utility,  and  cost- 
effectiveness. 

This  research  was  commissioned  by  Dr.  David  Hardy,  Director,  Air  Force  Research 
Faboratory  Directed  Energy  Directorate,  and  was  conducted  by  the  Force  Modernization  and 
Employment  Program  of  RAND  Project  AIR  FORCE.  This  report  represents  the  deliverable  to 
the  FY15  project,  “Assessing  the  Business  Case  for  the  Maui  High  Performance  Computing 
Center.”  This  report  should  be  of  interest  to  researchers  and  policymakers  who  are  thinking 
about  the  future  of  supercomputing  within  the  Department  of  Defense. 

RAND  Project  AIR  FORCE 

RAND  Project  AIR  FORCE  (PAF),  a  division  of  the  RAND  Corporation,  is  the  U.S.  Air  Force’s 
federally  funded  research  and  development  center  for  studies  and  analyses.  PAF  provides  the  Air 
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Force  with  independent  analyses  of  policy  alternatives  affecting  the  development,  employment, 
combat  readiness,  and  support  of  current  and  future  air,  space,  and  cyber  forces.  Research  is 
conducted  in  four  programs:  Force  Modernization  and  Employment;  Manpower,  Personnel,  and 
Training;  Resource  Management;  and  Strategy  and  Doctrine.  The  research  reported  here  was 
prepared  under  contract  FA7014-06-C-0001. 

Additional  information  about  PAF  is  available  on  our  website: 

http://www.rand.org/paf/ 

This  report  documents  work  originally  shared  with  the  U.S.  Air  Force  on  April  17,  2015.  The 
draft  report,  issued  on  May  26,  2015,  was  reviewed  by  formal  peer  reviewers  and  U.S.  Air  Force 
subject-matter  experts. 
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Summary 


Located  in  Kihei  on  the  island  of  Maui,  Hawaii,  the  Maui  High  Performance  Computing  Center 
(MHPCC)  was  established  in  1993  as  a  center  within  the  Department  of  Defense’s  (DoD)  High 
Performance  Computing  Modernization  Program  (HPCMP).  As  one  of  the  five  HPCMP  centers, 
MHPCC ’s  primary  mission  is  to  provide  computing  cycles  and  other  high  performance 
computing  (HPC)  capabilities  to  DoD’s  research,  development,  test  and  evaluation  (RDT&E) 
community.  DoD  scientists  and  engineers  from  around  the  world  utilize  MHPCC’ s  hardware, 
software,  and  technical  talent  to  develop  and  support  war-fighting  capabilities.  Currently, 
MHPCC ’s  largest  machine,  Riptide,  represents  3  percent  of  the  total  computing  cycles  across  the 
HPCMP  enterprise  in  support  of  this  broader  mission.1  MHPCC  is  managed  by  the  Directed 
Energy  Directorate  within  the  Air  Force  Research  Laboratory  (AFRL),  which  oversees  the  site’s 
annual  operating  budget  of  $14  million  and  supports  about  50  personnel. 

In  the  fall  2014,  Congress  asked  DoD  to  begin  considering  ways  to  cut  about  $45  million 
from  the  HPCMP  so  future  fiscal  years  could  be  supported  entirely  by  the  Army’s  programmed 
budget  of  $183  million.  Findings  within  this  report  are  designed  to  provide  AFRL 
decisionmakers  with  important  context  about  potential  future  missions  and  use  cases  for  MHPCC 
as  Congress,  the  Army,  and  the  HPCMP  work  toward  a  solution  for  resolving  the  budget  gap. 

To  help  provide  AFRL  with  this  context,  we  sought  to  answer  the  following  research 
questions:  Which  of  MHPCC ’s  capabilities  are  used  most  often  by  customers  in  the  Pacific  area 
of  responsibility  (AOR),  notably  U.S.  Pacific  Command  (PACOM)?  And,  specifically,  is  there  a 
demand  for  a  supercomputing  resource  that  is  located  within  the  AOR?  What  recommendations 
should  AFRL  consider  when  making  decisions  about  the  future  of  MHPCC? 

We  adopted  a  three-step  approach  to  gather  the  data  necessary  to  address  these  questions.  We 
began  by  reviewing  all  of  the  existing  documentation  about  MHPCC.  After  performing  this 
background  research,  we  also  met  with  HPC  experts  to  learn  about  trends  within  both  the 
government  and  commercial  markets.  As  a  final  step,  we  met  with  key  principals  within  the 
Pacific  AOR  to  learn  about  their  mission  needs  and  future  visions.  Our  goal  during  these 
interviews  was  to  determine  how  HPC  assets  are  currently  being  used  to  achieve  these  goals. 

After  gathering  all  of  these  data,  we  identified  common  themes  and  outliers,  and  we  made  a 
list  of  key  findings.  Specifically,  our  research  allowed  us  to  make  the  following  observations: 

•  There  is  a  consistent  lack  of  understanding  among  the  customer  base  as  to  what  problems 
are  suitable  for  a  supercomputer  or  HPC  assets,  in  general. 
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DoD  HPC  Modernization  Office  personnel,  phone  interview  with  author,  December  18,  2014. 
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•  While  several  entities  within  PACOM  offered  a  stated  preference  for  a  supercomputing 
facility  within  the  geographic  region,  we  were  unable  to  identify  any  entity  within  the 
Pacific  AOR  with  a  quantitative  requirement  for  real  time  supercomputing.  In  the 
absence  of  such  a  requirement,  we  did  gather  anecdotal  evidence  for  the  value  that 
PACOM  users  receive  by  having  HPC  expertise  within  the  AOR. 

•  PACOM  users  value  MHPCC’s  ability  to  stand  up  and  support  custom  servers.  These 
users  also  value  MHPCC’s  resident  expertise  in  programming  for  HPC  architectures. 

•  MHPCC  and  AFRL’s  Maui  optical  observatory  both  benefit  from  being  co-located. 

Based  on  these  findings,  our  research  resulted  in  the  following  six  recommendations  for  AFRL: 

•  Be  explicit  about  the  different  capabilities  resident  at  MHPCC.  Potential  users  need 
to  be  aware  of  what  MHPCC  has  to  offer,  what  reach  back  is  available  from  other  entities 
within  the  HPCMP  enterprise,  and,  if  appropriate,  what  other  services  are  available 
outside  the  HPCMP. 

•  MHPCC  should  market  itself  as  offering  three  separate  products:  (1)  a  large,  12,096- 
node  supercomputer;  (2)  a  facility  that  has  the  expertise  and  infrastructure  to  host 
standalone  HPC  assets;  and  (3)  a  research  staff  with  the  capacity  to  develop  creative 
solutions  to  operational  problems.  Presenting  MHPCC  as  a  center  that  offers  separate 
products  will  allow  for  a  more-tailored  approach  to  solving  customer  problems. 

•  Provide  PACOM  users  with  a  framework  for  matching  HPC  assets  to  the  desired 
workflow.  HPC  is  inherently  technical,  and  PACOM  customers  need  simple  frameworks 
to  understand  how  their  problems  fit  into  the  greater  context. 

•  Employ  both  strategic  and  technical  thought  leaders  to  enhance  Pacific  AOR 
customer  exploitation  of  MHPCC  capabilities.  Both  components  are  important.  The 
strategists  need  to  identify  areas  where  Pacific  AOR  users  might  benefit  from  HPC 
capabilities,  and  the  strategists  need  to  work  with  the  subject-matter  experts  to  help  them 
develop  future  requirements  for  HPC  assets. 

•  Rebuild  trust  and  understanding  with  the  HPCMP  leadership.  Reestablishing  trust 
always  takes  time,  but  MHPCC  management  can  take  some  initial  steps  by  setting  up 
consistent  communication  with  HPCMP  leadership. 

•  Recognize  that  aspects  of  the  computing  enterprise  are  changing.  Develop  a  strategy 
to  prepare  for  this  change.  As  computing  cycles  become  more  of  a  commodity,  it  will 
be  more  important  than  ever  to  develop  a  value  proposition  that  focuses  more  on 
MHPCC’s  human  capital  and  is  not  tied  to  hardware  alone. 
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1.  Introduction 


The  Department  of  Defense’s  (DoD)  High  Performance  Computing  Modernization  Program 
(HPCMP)  was  started  in  1992  with  the  purpose  of  developing  supercomputing  resources  and 
expertise  to  support  DoD’s  research,  development,  test,  and  evaluation  (RDT&E)  community. 
From  its  inception  to  201 1,  the  HPCMP  was  managed  by  the  Office  of  the  Secretary  of  Defense 
(OSD),  and  represented  a  $220  million  annual  program  element  within  the  OSD  budget.2 3 
Starting  in  fiscal  year  (FY)  2012,  HPCMP  was  transferred  to  the  Army,  which  assigned  the 
program  to  the  Army  Corps  of  Engineers  Research  and  Development  Center  for  oversight  and 
execution.  However,  this  transfer  from  OSD  to  the  Army  was  also  accompanied  by  a  reduction 
in  funds:  In  FY12,  the  program  was  provided  with  $183  million,  a  $37  million  shortfall  when 
compared  with  the  previous  fiscal  year.4  Since  then,  Congress  has  increased  funding  each  year 
over  the  President’s  request  by  approximately  $45  million  to  ensure  continuity  with  pre-FY12 
funding  levels. 

In  fall  2014,  Congress  asked  DoD  to  begin  thinking  about  a  longer-term  solution  to  this 
problem.  Specifically,  DoD  was  asked  to  consider  the  effect  on  HPCMP  if  it  was  funded  only  at 
the  President’s  budget  request  of  approximately  $185  million  a  year.  In  addition,  Congress  asked 
for  a  strategy  to  close  this  gap  in  future  years. 

In  response  to  this  request,  the  Army  submitted  a  report  to  Congress  in  January  2015  that 
summarized  the  high  performing  computing  (HPC)  program  and  highlighted  its  value  to  the 
entire  defense  enterprise.  The  report  also  lays  out  a  selection  of  courses  of  action  that 
decisionmakers  can  take  in  order  to  close  the  $45  million  budget  gap.  The  options  range  from 
eliminating  two  of  the  program’s  five  supercomputers  to  reducing  research  investments  in 
network  and  software  development  to  eliminating  30  Ph.D. -level  experts  currently  in  residence  at 
DoD  labs.5  As  of  the  date  of  this  report,  Congress  has  not  yet  responded  with  a  budget  decision. 

Research  Objective  and  Approach 

In  light  of  this  background,  the  Air  Force  Research  Faboratory  (AFRF)  asked  RAND  to  evaluate 
the  capabilities,  challenges,  and  future  options  for  one  of  the  HPCMP’s  supercomputing  sites  it 
manages,  the  Maui  High  Performance  Computing  Center  (MHPCC).  This  report,  which 

2 

Department  of  the  Army,  Report  to  Congress:  High  Performance  Computing  Modernization,  Headquarters , 
January  2015. 

3 

Air  Force  Research  Laboratory/Research  and  Development  personnel,  phone  conversation  with  author,  June  3, 
2014. 

4 

Department  of  the  Army,  2015. 

5  Department  of  the  Army,  2015. 
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summarizes  our  research  method,  key  findings,  and  recommendations,  is  designed  to  provide 
AFRL  decisionmakers  with  important  context  about  MHPCC  as  Congress,  the  Army,  and  the 
HPCMP  work  toward  resolving  the  budget  gap. 

To  help  meet  this  objective,  we  sought  to  answer  the  following  research  questions:  Which  of 
MHPCC ’s  capabilities  are  used  most  often  by  customers  in  the  Pacific  area  of  responsibility 
(AOR),  notably  U.S.  Pacific  Command  (PACOM)?  And,  specifically,  is  there  a  demand  for  a 
supercomputing  resource  that  is  located  within  the  AOR?  What  recommendations  should  AFRL 
consider  when  making  decisions  about  the  future  of  MHPCC? 

We  adopted  a  three-step  approach  to  gather  the  data  necessary  to  address  these  questions.  We 
began  the  project  by  searching  for  and  reviewing  existing  documentation  about  the  MHPCC,  its 
capabilities,  and  the  customer  base.6  We  paid  particular  attention  to  materials  that  provided 
historical  context  about  how  HPCMP  and  MHPCC  got  to  where  they  are  today.  We  also  met 
with  experts  both  internal  and  external  to  RAND  to  learn  about  the  trends  within  the  HPC 
community  and  the  broader  computing  industry.7 * 

Once  we  understood  the  basics  about  the  site’s  capabilities  and  HPC  in  general,  we  met  with 
MHPCC  personnel  to  gain  a  more-nuanced  understanding  about  the  site’s  existing  resources, 

O 

customers,  and  current  challenges.  We  also  met  with  HPCMP  office  personnel  to  get  their 
perspective  on  the  entire  program,  along  with  how  MHPCC  fit  into  it.9 

As  a  final  step  in  the  data-gathering  process,  we  met  with  key  principals  within  the  Pacific 
AOR  to  ask  about  their  current  mission  needs,  along  with  their  future  vision  and  strategy  for 
achieving  that  vision.  Our  goal  during  these  conversations  was  to  help  identify  how  HPC  assets 
are  currently  used — or  might  be  used  in  the  future — to  achieve  these  goals.  We  were  also 
interested  in  ways  AFRL  could  derive  value  from  MHPCC  beyond  its  HPCMP ’s  RDT&E 
mandate.  To  that  end,  we  had  interactions  with  both  senior  leaders  and  technical  experts  within 
Pacific  Air  Forces/ A2,  United  States  Pacific  Command/J8,  United  States  Pacific  Command/J9, 


6  Charles  C.  Holland,  Maui  Report ,  Pittsburgh,  Pa.:  Carnegie  Mellon  University,  July  2012;  DoD  High  Performance 
Computing  Modernization  Program,  “DoD  High  Performance  Computing  Modernization  Program:  Portal  Program,” 
presented  May  13,  2014a,  not  available  to  the  general  public;  Clifford  E.  Rhoades,  “Defense  Research  and 
Engineering  Network  (DREN)  Hawaii  Overview,”  for  Hawaii  Intranet  Consortium,  presented  January  20-22,  2015, 
not  available  to  the  general  public;  MHPCC/DSRC,  Quarterly  Status  Report,  July  1-September  30,  2014a,  not 
available  to  the  general  public;  DoD  HPCMP,  CREATE  Quarterly  Update ,  October  2014b,  not  available  to  the 
general  public;  DoD  HPC  Modernization  Office,  Insights ,  fall  2014;  AFRL-MHPCC,  “Top  10  Projects — FYTD14,” 
Excel  spreadsheet  received  via  email  on  November  3,  2014,  not  available  to  the  general  public. 

7 

Jim  Brase,  in-person  discussion  with  the  author  at  the  RAND  Corporation,  Santa  Monica,  Calif.,  December  5, 
2014;  Jim  Brase,  phone  discussion  with  the  author  from  Santa  Monica,  Calif.,  to  Lawrence  Livermore  National 
Laboratory,  March  3,  2015;  AFRL  personnel,  in-person  discussion  with  the  author  in  Arlington,  Va.,  December  11, 
2014;  DoD  HPCMP  personnel,  phone  discussion  with  the  author,  February  15,  2015;  Lawrence  Livermore  National 
Laboratory  personnel,  phone  discussion  with  author,  February  26,  2015. 

g 

MHPCC  personnel,  phone  discussion  with  author,  July  14,  2014a;  MHPCC  personnel,  in-person  discussion  with 
author,  October  27-28,  2014b. 

9 

DoD  HPC  Modernization  Office  personnel,  phone  interview  with  author,  December  18,  2014;  DoD  HPCMP 
personnel,  2015. 
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the  AFRL  optical  observatory  on  Maui,  the  Pacific  Missile  Range  Facility  (PMRF),  and  the 
Defense  Intelligence  Agency.10 

After  the  data-gathering  phase  of  the  research,  we  summarized  the  key  points  from  each 
discussion,  and  we  started  looking  for  common  themes  and  outliers.  We  compiled  a  list  of  key 
findings,  which  are  summarized  in  this  report.  However,  we  first  need  to  provide  some  additional 
context  on  nomenclature,  MHPCC,  and  two  key  factors  that  set  some  boundaries  for  MHPCC’s 
future  options. 

Supercomputing  Defined 

One  of  the  challenges  of  working  on  this  topic  is  that  there  are  different  lexicons  used  to  describe 
HPC.  For  the  purpose  of  this  report,  we  define  three  categories  of  computing  assets:  standalone 
computers,  cluster  computers,  and  supercomputers. 

We  define  standalone  machines  as  conventional  commercial-off-the-shelf  (COTS)  boxes  that 
are  deployed  in  standard  business,  scientific,  or  personal  settings.  Standalone  machines  can 
certainly  be  very  powerful,  but  their  defining  characteristic  is  that  they  are  off-the-shelf 
consumer  parts  that  utilize  existing  software  packages  for  solving  general  computing  tasks. 

We  will  use  the  term  cluster  computer  to  refer  to  loosely  coupled  collections  of  computers — 
typically  COTS  grade  or  higher — that  are  architected  to  work  on  computational  problems  in 
concert.  Clusters  are  best  suited  for  problems  with  subparts  that  can  be  solved  in  parallel,  for 
example,  when  searching  a  large  database  for  records  satisfying  a  particular  condition.  To  solve 
this,  the  database  can  be  divided  into  smaller  shards  that  are  distributed  to  many  nodes,  and  each 
node  performs  the  search  in  parallel  with  the  rest.  The  desired  records  are  then  collected  after  the 
parallel  searches  are  completed.  However,  during  the  time  that  the  search  is  being  performed  on 
each  shard,  none  of  the  nodes  has  reason  to  communicate  with  one  another. 

We  based  our  use  of  the  term  supercomputer  on  a  convention  set  by  Dongarra  and  reflecting 
trends  highlighted  by  Bell  in  the  ranks  of  top-performing  HPC  assets.* 11  Supercomputers  are 


10  Pacific  Air  Forces  (PACAF)  A2  personnel,  in-person  discussion  with  author,  October  27,  2014;  PACOM/J8 
personnel,  in-person  discussion  with  author,  October  27,  2014;  PACOM/J8  personnel,  phone  conversation  with 
author,  March  12,  2015;  613  AOC  personnel,  in-person  discussion  with  author,  October  27,  2014;  Defense 
Intelligence  Agency  personnel,  in-person  discussion  with  author,  October  27,  2014;  PACOM/J9  personnel,  in- 
person  discussion  with  author,  October  27,  2014;  Pacific  Missile  Test  Range  personnel,  in-person  discussion  with 
author,  October  28,  2014;  Pacific  Missile  Test  Range  personnel,  in-person  discussion  with  author,  January  26,  2015; 
Pacific  Missile  Test  Range  range  safety  personnel,  phone  conversation  with  author,  March  3,  2015;  PACOM/J8I 
personnel,  phone  conversation  with  author,  January  30,  2015;  AFRL/RDSM  personnel,  ongoing  email  discussion 
with  authors,  March  30-April  16,  2015c. 

11  Gordon  Bell  and  Jim  Gray,  “What’s  Next  in  High-Performance  Computing?”  Communications  of  the  ACM , 

Vol.  45,  No.  2,  2002,  pp.  91-95;  Jack  Dongarra,  Thomas  Sterling,  Horst  Simon,  and  Erich  Strohmaier,  “High- 
Performance  Computing:  Clusters,  Constellations,  MPPs,  and  Future  Directions,”  Computing  in  Science  and 
Engineering ,  Vol.  7,  No.  2,  2005,  pp.  51-59. 
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custom-designed  integrated  systems  made  up  of  a  tight-knit  collection  of  top-grade  cores. 
Supercomputers  include  low-latency  parallel  clusters,  massively  parallel  processors  (MPPs),  and 
some  descendants  of  vector  processing  supercomputers. 

Supercomputers  excel  at  problems  where  subtasks  need  to  be  fed  back  into  the  computation. 
Physics-based  simulations  are  the  most-common  example  of  this  type  of  problem:  As  the  model 
run  evolves,  the  results  from  the  Nth  iteration  will  serve  as  the  boundary  conditions  for  the 
{N+l  }th  iteration.  To  address  this  need,  the  computing  architecture  must  offer  low  latency 
between  the  processor  and  memory  elements  within  the  system,  so  supercomputers  (by  our 
definition)  are  characterized  by  the  use  of  high-speed,  low-latency  interconnects  among  the 
processor  and  memory  elements. 

To  our  knowledge,  most  assets  within  the  HPCMP  enterprise  can  be  classified  as  standalone 
machines  and  supercomputers,  using  this  definition.  However,  in  order  to  avoid  confusion,  we 
will  use  the  term  high  performance  computing  assets  as  a  generic  way  to  describe  the  HPCMP ’s 
computing  resources  when  we  lack  the  technical  specifications  needed  to  label  them  with  a 
specific  category. 

In  practice,  recent  non-standalone  assets  often  fall  on  a  spectrum  between  cluster  computers 
and  supercomputing.  This  reflects  rapid  developments  in  the  HPC  market  in  response  to  an 
emerging  unified  demand  for  both  kinds  of  capability.  A  lot  of  modem  scientific  data  analyses 
(e.g.,  for  CERN’s  Large  Hadron  Collider,  searches  on  large  genomic  databases)  require  raw 
data-analytic  power  typically  found  in  cluster  computers.  There  is  a  pressing  need  for  assets  that 
can  apply  complex  iterative  algorithms  to  manage  the  flood14  of  new  data  from  ubiquitous 
sensors.  Thus,  the  asset  classes  highlighted  here  are  best  thought  of  more  as  extremes  on  a 
spectrum  than  as  categorical  definitions. 

Some  additional  context  related  to  these  definitions  is  shown  in  Table  1.1. 


12 

A  core  refers  to  a  low-level  data-processing  element  capable  of  executing  instructions  independently.  This 
includes  traditional  cores  on  multithreaded  central  processing  units  (CPUs),  graphics  processing  unit  cores,  and 
other  programmable  hardware  or  virtual  implementation.  Core  count  is  a  useful  index  of  how  many  processes  a 
system  can  run  concurrently. 

13 

Michael,  Schmelling,  Markward  Britsch,  Nikolai  Gagunashvili,  et  al.,  “RAVEN-Boo sting  Data  Analysis  for  the 
LHC  Experiments,”  Applied  Parallel  and  Scientific  Computing ,  Vol.  7134,  2010,  pp.  206-214. 

14 

Isaac  R.  Porche,  Bradley  Wilson,  Erin-Elizabeth  Johnson,  Shane  Tierney  and  Evan  Saltzman,  Data  Flood: 
Helping  the  Navy  Address  the  Rising  Tide  of  Sensor  Information,  Santa  Monica,  Calif.:  RAND  Corporation,  RR- 
3 15-NAVY,  2014. 
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Table  1.1.  Comparison  of  Different  Computing  Assets 


Standalone 

Cluster  Computer 

Supercomputer 

Typical  Data 

Load 

Local. 

Smaller  data  sets 
<10GB. 

Distributed. 

Larger  data  sets  GBs  to 
100s  of  TB 

Distributed. 

Larger  data  sets  GBs  to  PBs. 

Parallelizability 

Mostly 

sequential. 

Minimal 
multithreading 
(1-16  threads) 
with  poor 
scalability. 

Highly  parallelized  on  up 
to  lOOKs  of  compute 
nodes.  Often  high- 
latency  between  nodes. 
Highly  scalable  (often 
dynamically  so). 

Sequential  +  parallel 
workloads.  Often  parallelized 
at  the  processor  level.  Low 
latency  among  100,000s  of 
cores. 

Responsiveness 

Small  fast 
systems,  suitable 
for  small  real¬ 
time  processing 

Large  scale  can  cause 
slow  response.  Mostly 
batch  processing. 

Large  scale  but  with  fast, 
custom  interconnects.  Low 
latency,  fast  response. 

Software 

Ecosystem 

Large  market  of 
COTS  software. 
Large  expertise 
pool  for  custom 
development. 

Dominated  by  a  smaller 
number  of  large-scale 
commercial  solutions 
(Hadoop/Spark). 

Lots  of  open  source 
code.  Development 
expertise  plenty.  Some 
software  monopolies. 

Lots  of  small-scale 
commercial  and  bespoke 
software  (MPI/Coarray 
FORTRAN).  Custom 
hardware  demands 
specialized/rare  development 
tools  and  expertise. 

Typical  use  cases 

Watching  a  DVD 
(-1.4GB). 

Analyses  of  3-D 
medical  images 
(-10GB) 

Indexing  and  ranking  “all” 
worldwide  web  pages 
(~30  trillion  pages). 

Training  a  movie 
recommendation  system 
using  past  preferences  of 
millions  of  users  (~40 
million  users). 

Real-world  simulation  of 
airflow  and  turbulence  for  new 
aircraft  designs. 

Simulation  of  in-water  wave 
and  keel  behavior  for  sea 
vessels  in  design. 

Simulation  of  flames  and 
explosions  for  fuel  safety 
research. 

HPC  assets  are  crucial  to  U.S.  defense  and  technology  leadership.  The  DoD  acquisition 
program  relies  heavily  on  computer  simulations  run  on  HPCs  administered  through  the  HPCMP 
(formerly  under  OSD,  now  under  the  Army).  They  are  important  at  each  RDT&E  phase  in  the 
development  of  frontier  technology  (e.g.,  jet  engines  for  the  Air  Force).  HPCs  also  help 
guarantee  the  efficacy  of  the  U.S.  nuclear  weapons  stockpile — helping  to  maintain  the  U.S. 
nuclear  deterrence  posture  in  spite  of  test  ban  treaties  in  force.  And,  going  forward,  HPCs  can  be 
key  assets  for  effectively  managing  the  massive  amounts  of  data  involved  in  cyberwarfare. 
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2.  MHPCC  Overview 


Located  in  Kihei,  on  the  island  of  Maui,  Hawaii,  the  MHPCC  occupies  two  physical  locations. 
We  will  refer  to  the  main  HPC  Center  as  Building  90,  and  it  contains  the  MHPCC’s  largest  HPC 
asset.  The  second  facility  is  nearby  (Building  50),  which  contains  offices  for  the  administrative 
and  technical  staff,  along  with  secure  areas  for  classified  workstations.  MHPCC  also  operates 
and  sustains  two  small  HPC  assets.15  In  addition  to  MHPCC  personnel,  the  administrative 
building  also  houses  the  staff  and  some  support  functions  for  AFRL’s  optical  observatory,  which 
is  located  on  top  of  nearby  Haleakala  volcano. 

For  FY15,  MHPCC  has  a  projected  operating  budget  of  $14. 1  million  and  supports  about  55 
personnel.16  The  total  operating  expenses  are  shown  in  more  detail  in  Figure  2.1.  The  single 
largest  expense  is  the  support  of  the  HPC  Center.  Half  of  this  $5.0  million  is  used  to  pay  for 
labor;  the  rest  is  for  electricity  ($1.1  million),  rent  ($0.9  million),  and  the  supercomputer’s 
warranty  support  ($0.5  million).  The  next  largest  expense  is  $3.6  million  for  common  shared 
services  that  support  software  development  and  HPC  Center  operations,  which  includes  site 
management,  office  equipment,  financial  reporting,  utility  servers,  the  local  help  desk,  water, 
networking,  and  outreach.  The  administration  building  (Building  50)  costs  about  $2.6  million, 
and  we  list  this  in  Figure  2.1  as  “combined  support  to  AFRL/MHPCC,”  because  both 
organizations  use  this  facility.  (AFRL  pays  a  commensurate  portion  of  this  cost  because  its 
observatory  personnel  reside  in  this  building.)  This  $2.6  million  covers  the  administration 
building’s  lease  ($1.0  million)  and  electricity  ($0.6  million),  along  with  security  costs  and  facility 
maintenance.  The  remaining  slice  in  Figure  2.1  is  $2.9  million,  which  is  devoted  to  supporting 
software  development  for  HPCMP.  This  software  development  capability  is  enterprisewide  in 
support  of  the  HPCMP,  and  it  is  located  at  MHPCC. 


15  AFRL/RDSM  personnel,  2015c. 

16  MHPCC  DSRC,  “Organizational  Chart  as  of  October  15,  2014,”  received  via  email,  November  18,  2014c,  not 
available  to  the  general  public;  AFRL/RDSM  personnel,  email  correspondence  with  author,  March  18-31,  2015a; 
AFRL/RDSM  personnel,  “Maui  Top  Line  Budget  Information,”  received  via  email,  March  19,  2015b,  not  available 
to  the  general  public. 
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Figure  2.1.  MHPCC  Budget  Overview  ($14.1  M) 


$5.0M 

(36%) 


$3.6M  (25%) 


i  Common  Support 
Services 

Software  Development 


Combined  Support  to 
AFRL/MHPCC 

HPC  Center  Expenses 


_  $2.9M  (20%) 

$2.6M  (19%) 

SOURCE:  Data  from  MHPCC  DSRC  and  AFRL/RDSM. 


MHPCC  supports  a  workforce  of  55  full-time  equivalents  (FTE).  Figure  2.2  shows  more 
details  on  how  these  resources  are  used;  and  the  result  aligns  closely  with  the  financial  support 
previously  outlined.  The  largest  group  (19  FTE)  works  in  support  of  the  HPC  Center,  and  the 
second  largest  group  (18  FTE)  represents  general  administrative  duties,  such  as  human  resources, 
communications,  external  affairs,  and  site  management.  Eleven  FTE  support  the  HPCMP’s 
Portal  and  CREATE  programs  (we  group  these  two  programs  together  as  “Software 
Development”),  and  seven  FTE  are  devoted  to  security  and  facility  upkeep.17 


Figure  2.2.  MHPCC  Personnel  Overview  (Total  FTE  =  55) 


(20%) 


■  General  Administrative 

■  Software  Development 

■  Security,  Facilities,  etc. 

■  HPC  Center  Expenses 


SOURCE:  Data  from  MHPCC  DSRC  and  AFRL/RDSM. 


17  AFRL/RDSM  personnel,  2015b;  AFRL/RDSM  personnel,  “MHPCC  Cost  Breakdown,”  received  via  email,  April 
15,  2015 e,  not  available  to  the  general  public. 
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The  site  has  a  variety  of  computing  resources,  ranging  from  a  large  supercomputer  to  a 
number  of  smaller  servers  that  are  stationed  at  different  classification  levels.  Riptide,  MHPCC’s 
largest  machine,  is  a  12,096-core  supercomputer  that  was  most  recently  ranked  286  in  TOP500’s 
list  of  the  world's  fastest  computers.  In  addition  to  Riptide,  MHPCC  houses  and  maintains  HPC 
assets  for  other  DoD  customers  within  PACOM’s  AOR.  Finally,  MHPCC  provides  floor  space — 
but  does  not  own  or  pay  for  operation  and  maintenance — for  a  large  image -processing 
supercomputer  for  the  University  of  Hawaii’s  Pan-STARRS  optical  observatory.19 

Complementing  the  physical  hardware,  MHPCC  also  represents  a  key  node  on  the  HPCMP’s 
proprietary  network,  the  Defense  Research  and  Engineering  Network  (DREN).  The  DREN  offers 
fast,  10-gigabit-per-second  Ethernet  connectivity  to  the  rest  of  the  HPCMP  enterprise,  and  it  also 

90 

offers  a  test  bed  for  research  focused  on  improving  network  bandwidth. 

Two  Factors  Set  Boundaries  for  MHPCC’s  Future 

There  are  two  preexisting  conditions  that  will  have  a  big  impact  on  MHPCC’s  future:  the  high 
cost  of  electricity  and  the  effects  from  a  series  of  recent  contracting  and  program  management 
missteps.  Together,  these  factors  impose  some  boundary  conditions  on  the  site’s  future  options, 
so  it  is  worth  explaining  them. 

Electricity  on  Maui  is  expensive.  Roughly  70  percent  of  the  island’s  electrical  capacity  is 
generated  by  burning  oil,  all  of  which  has  to  be  shipped  in  from  distant  locations.  Because  of 

this,  a  kilowatt-hour  of  electricity  on  Maui  in  2013  averaged  36.5  cents,  about  five  times  the  rate 

22 

on  the  U.S.  mainland. 

With  its  12,096  cores,  Riptide  draws  about  200  kilowatt  hours  (kWh)  on  average,  and  this 
translates  to  about  $75  per  hour  in  electricity  costs  when  the  machine  is  running  at  a  nominal 
rate.  By  contrast,  HPCMP’s  other  four  supercomputing  sites  are  on  the  U.S.  mainland,  where 
power  is  considerably  less  expensive.  The  price  at  the  other  sites  range  from  6  cents  to  7.5  cents 
per  kWh.24  If  Riptide  was  at  one  of  these  facilities,  the  same  hour  of  electricity  would  cost  $12 
to  $15  per  kWh,  depending  on  the  continental  U.S.  location. 


1 8 

Ranking  was  286  as  of  November  2014.  See  TOP500,  “TOP500  Supercomputing  Sites:  Maui  High-Performance 
Center  (MHPCC),”  web  page,  undated. 

19 

MHPCC  DSRC,  “Computing  Resources,”  received  via  email,  November  2014b,  not  available  to  the  general 
public. 

20 

Rhoades,  2015. 

21 

The  Hawaiian  Electric  Companies,  “Power  Facts,”  web  page,  undated. 

22 

AFRL/RDSM  personnel,  “DSRC  Sites  Power  Forecast,”  received  via  email,  April  2,  2015d,  not  available  to  the 
general  public. 

23  AFRL/RDSM  personnel,  2015c. 

24 

AFRL/RDSM  personnel,  2015d. 
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This  high  cost  of  electricity  imposes  a  practical  requirement  on  the  site’s  value  proposition. 
This  means  that  any  supercomputing  job  within  the  Pacific  AOR  that  can  afford  the  latency  of 
sending  the  job  back  to  the  U.S.  mainland  should  do  so,  because  this  will  usually  represent  the 
least-expensive  solution. 

The  exception  to  this  rule  is  when  the  results  from  a  job  are  needed  in  near-real  time,  and  the 
user  cannot  afford  to  wait  for  the  delay  associated  with  sending  the  execution  command,  the 
supporting  dataset,  or  both,  to  the  mainland  and  back.  Thus,  there  is  one  type  of  job  within  the 
Pacific  AOR  where  it  might  make  sense  to  spend  a  little  more  on  MHPCC,  and  those  are  the  jobs 
where  the  computation  needs  to  happen  in  near-real  time.  For  the  remainder  of  this  report,  we 
will  refer  to  these  jobs  as  “real-time  jobs,”  which  we  consider  to  be  the  opposite  of  “batch  jobs.” 
Real-time  jobs  are  those  whose  results  are  highly  perishable;  high-frequency  stock  trading  is  a 
classic  example  from  the  civilian  world.  Batch  jobs  are  calculations  whose  results  remain  valid 
for  a  measurable  period  of  time;  some  examples  are  modeling  the  fluid  dynamics  of  an  airfoil  or 
predicting  the  blast  radius  of  an  explosion. 

The  importance  of  MHPCC  and  real-time  jobs  was  a  key  focus  point  during  our  stakeholder 
interviews.  When  we  spoke  with  the  community  within  the  Pacific  AOR,  we  continually  pressed 
users  and  potential  users  on  whether  their  computing  jobs  needed  to  be  performed  in  real  time,  or 
whether  the  jobs  could  be  queued  in  advance  and  the  answer  reported  after  the  jobs  had  been  run 
as  a  batch.  We  were  particularly  interested  in  this  because  any  requirement  for  real-time 
calculations  in  the  Pacific  AOR  represented  a  unique  opportunity  that  could  only  be  met  with  a 
supercomputer  in  the  Pacific. 

The  second  initial  condition  that  will  impose  informal  restrictions  on  MHPCC’ s  future  is 
related  to  contracting  and  program  management  challenges  that  have  led  to  tension  between 
MHPCC  and  the  HPCMP.  Our  research  uncovered  three  recent  incidents  worth  noting. 

The  first  example  is  related  to  a  project  that  was  intended  to  reduce  the  price  that  MHPCC 
pays  for  electricity.  In  FY10,  MHPCC  requested  and  was  awarded  $14  million  from  the  HPCMP 
to  build  a  small  solar  farm  near  MHPCC  that  would  reduce  the  cost  of  electricity  from 
$0.35  kWh  to  $0.15  kWh.25  In  FY12,  MHPCC  had  to  return  $12  million  because  the  funds  were 
left  on  an  expired  contract  and  were  not  recycled  in  time.  Eventually,  it  was  determined  that  the 
funds  were  returned  to  the  HPCMP,  reallocated  to  other  projects,  and  the  solar  project  was  scaled 
back.26 

The  site  has  also  faced  challenges  with  a  201 1 -issued  request  for  proposal  for  the  cost-plus- 
fixed-fee  contract  to  operate  the  MHPCC.  This  contracting  decision  has  been  through  two 
successive  protests,  with  the  U.S.  Government  Accountability  Office  ultimately  deciding  in  June 


25 

AFRL/RD  personnel,  phone  conversation  with  author,  June  3,  2014;  DoD  HPC  Modernization  Office  personnel, 
2014. 

26  AFRL/RDSM  personnel,  2015c. 
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2014  that  the  Air  Force  should  reimburse  two  of  the  bidders  for  the  costs  associated  with  filing 

97 

and  pursuing  their  protests. 

As  of  the  date  of  this  report,  this  contract  has  been  canceled  and  MHPCC  is  currently 
operating  under  a  sole  source  transition  contract  via  AFRL.  While  this  arrangement  will  provide 
continuity  of  operations  for  MHPCC,  the  HPCMP  expressed  frustration  that  it  had  to  modify  one 

98 

of  its  own  procurement  efforts  to  accommodate  this  change. 

Finally,  in  FY14,  there  was  an  issue  regarding  the  lease  of  the  MHPCC  administration 
building  (Building  50).  The  building  is  owned  by  the  University  of  New  Mexico  and  leased  to 
the  AFRL  observatory  (they  pay  quarterly  rent  in  arrears).  In  FY14,  the  previous  occupancy 
agreement  expired,  and  there  was  a  delay  in  payment  for  seven  months  because  the  $1  million 
lease  was  above  the  amount  that  automatically  triggers  a  congressional  notification.  Responding 
to  the  actions  from  this  trigger  required  coordination  between  the  lessor,  the  funding  agency 
(U.S.  Army),  and  the  occupant  (MHPCC/ AFRL).  The  lease  remained  unpaid  for  seven  months 
while  the  parties  determined  a  way  forward.  Eventually,  the  problem  was  resolved,  and  the  lease 
was  paid  in  full  through  January  2015.  AFRL  remains  current  on  its  lease  commitment  as  of  the 
date  of  this  report.30 

While  the  three  examples  stated  above  are  complicated  and  likely  not  entirely  the  fault  of 
MHPCC,  the  HPCMP  reported  to  us  that  it  has  been  “very  reluctant”  to  make  additional 
investments  in  the  site  because  it  sees  MHPCC ’s  approach  to  program  management  as  “high 
risk.”  As  a  result,  HPCMP  reported  that  it  is  not  planning  to  make  any  additional  significant 

O  1 

investments  in  infrastructure  at  MHPCC  unless  mandated  by  Congress.  Because  of  this,  the 
aging  Riptide  machine — which  was  once  DoD’s  second-most  powerful  computer  when  it  was 
installed  in  summer  2013 — is  not  slated  to  receive  a  biennial  replacement,  as  is  custom  within 
the  HPCMP.  Currently,  Riptide  represents  only  3  percent  of  the  total  computing  cycles  across  the 
HPCMP  enterprise,  and  that  figure  will  decrease  to  1  percent  when  biennial  upgrades  at  other 
HPCMP  sites  are  completed  by  the  end  of  FY15.32 

Both  of  these  initial  conditions — the  high  cost  of  electricity  and  the  series  of  recent 
contracting  and  program  management  missteps — have  led  to  some  initial  bounding  conditions. 
For  calculations  that  require  supercomputing,  the  task  will  only  be  cost-effective  (for  HPCMP)  to 
run  at  MHPCC  if  the  decisionmaker  resides  in  the  Pacific  AOR  and  has  a  need  for  real-time 


27 

Government  Accountability  Office,  VSE  Corporation;  The  University  of  Hawaii-Costs ,  B-407164.11, 
B-407164.12,  June  23,  2014;  Keoki  Kerr,  “Lost  UH  Supercomputing  Center  Contract  Worth  at  Least  $70M,” 
Hawaiian  News  Now ,  August  30,  2012. 

DoD  HPC  Modernization  Office  personnel,  2014;  AFRL/RDSM  personnel,  2015c. 

29 

AFRL/RDSM  personnel,  2015c. 

20 

AFRL/RDSM  personnel,  2015c. 

31  DoD  HPC  Modernization  Office  personnel,  2014. 

32  DoD  HPC  Modernization  Office  personnel,  2014. 
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calculations  (versus  batch  jobs).  In  addition,  MHPCC  currently  is  not  scheduled  to  receive  any 
significant  supercomputing  hardware,  and  soon  it  will  not  be  in  a  position  to  market  itself  as 
possessing  one  of  the  world’s  fastest  500  machines. 

These  two  MHPCC-specific  restrictions  negatively  impact  MHPCC ’s  value  proposition, 
especially  within  HPCMP’s  immediate  RDT&E  mandate.  So  we  visited  decisionmakers  within 
the  Pacific  AOR  on  AFRL’s  request  to  determine  how  MHPCC  might  be  able  to  add  value  to 
current  or  future  missions.  Our  findings  from  these  discussions  are  highlighted  in  Chapter  Three. 
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3.  Key  Findings 


PACOM  Customers  View  Supercomputing  as  “Anything  Larger  Than  a 
Desktop” 

The  first  observation  that  we  made  in  our  interviews  with  potential  PACOM  users  was  a 
consistent  lack  of  understanding  as  to  what  constituted  a  supercomputer.  In  fact,  for  many  users, 
a  supercomputer  was  “anything  larger  than  my  desktop,”  as  one  user  told  us.  In  addition,  most 
people  were  unaware  of  the  distinction  between  tasks  suited  for  cluster  HPCs  (highly  parallel 
tasks  like  pattern  identification)  or  a  supercomputer  (scientific  computing  problems  like  physics- 
based  models). 

The  client  may  not  need  to  understand  all  the  technical  details  of  MHPCC’s  assets.  But  a  lack 
of  understanding  of  key  asset  capabilities  undermines  MHPCC’s  ability  to  build  a  strong 
customer  base,  because  the  customers  do  not  understand  enough  about  HPC  to  incorporate  the 
capabilities  into  their  vision  statements.  This  problem  is  easily  addressed  with  a  refined  approach 
by  MHPCC’s  business  development  team,  and  we  outline  a  strategy  for  solving  this  problem  in 
the  final  chapter  of  this  report. 

No  Quantitative  Requirements  from  Pacific  AOR  Users  for  Supercomputing 
in  the  Pacific 

We  were  unable  to  identify  any  entities  within  the  Pacific  AOR  with  a  quantified  requirement  for 
real-time  supercomputing.  In  this  context,  we  specifically  use  the  word  supercomputing  to  refer 
to  HPCMP’s  largest  machines,  such  as  Riptide  at  MHPCC. 

In  the  absence  of  numerical  requirements,  some  of  the  stakeholders  were  able  to 
communicate  future  plans  that  may  require  real-time  supercomputing  in  the  future: 

•  PACOM/J9  is  working  on  an  initiative  for  using  the  Hawaiian  Islands  as  a  test  bed  for 
improving  infrastructure  resilience  in  isolated  geographies. 

•  PACOM/J8  is  interested  in  processing  the  real-time  data  streams  from  networks  of 
unmanned  underwater  vehicles  (UUVs).34 


33  PACOM/J9  personnel,  2014;  PACOM,  correspondence  from  Randall  Cieslak  to  John  West,  Camp  HM  Smith, 
Hawaii,  June  17,  2014. 

34 

PACOM/J8  personnel,  2014;  PACOM/J8  personnel,  2015;  PACOM,  correspondence  from  George  Kailiwai  to 
John  West,  Camp  HM  Smith,  Hawaii,  June  16,  2014. 
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•  PMRF  indicated  that  it  was  likely  to  need  real-time  computing  support  for  its  live  missile 
tests  “in  ten  years,”  but  it  has  not  started  thinking  about  the  requirements  needed  to 
support  such  an  activity. 

However,  when  pressed,  none  of  the  stakeholders  that  we  spoke  with  could  point  to  existing 
(or  even  planned)  quantified  requirements  for  real-time  supercomputing  that  would  necessitate 
the  presence  of  any  HPC  assets  within  the  region  (for  context,  the  geographic  locations  for  all 
DoD  Supercomputing  Resource  Centers  are  shown  in  Figure  3.1).  This  finding  suggests  that — 
based  on  our  interviews,  at  least — there  is  currently  not  enough  demand  from  Pacific  AOR  users 
to  necessitate  a  high-cost,  Riptide-like  machine  within  the  region. 

In  fact,  in  all  of  our  discussions  with  the  customer  base,  we  did  not  encounter  anyone  within 
the  Pacific  AOR  who  had  projects  running  on  MHPCC’s  Riptide  machine.  This  specific  finding 
is  not  surprising  because  of  the  way  that  HPC  assets  are  scheduled  and  tasked:  For  new  project 
ideas,  researchers  will  submit  proposals,  and  the  HPCMP  will  oversee  a  selection  process  to 
identify  qualified  applicants  and  eventually  award  hours  on  an  HPC  asset.  This  process  means 
that  significant  portions  of  the  jobs  that  run  on  Riptide  are  unrelated  to  the  PACOM  mission.  As 
an  example,  the  top  ten  MHPCC  projects  in  FY14  used  45  million  CPU  hours,  and  none  were 
tasked  directly  by  PACOM.  Of  these  projects,  one  managed  by  the  Defense  Threat  Reduction 
Agency  was  related  to  a  PACOM  mission  objective.  It  represented  0.1  percent  of  the  45  million 
CPU  hours.36 


35 

Pacific  Missile  Test  Range  personnel,  2015. 

36  AFRL-MHPCC,  2014. 
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Figure  3.1.  Geographical  Locations  of  Current  DoD  Supercomputing  Resource  Centers,  Including 
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SOURCE:  Data  from  DoD  HPC. 

PACOM  Users  Value  MHPCC’s  Support  for  Custom  Servers  and 
Programming  Expertise 

If  DoD  users  within  the  Pacific  AOR  are  not  running  jobs  on  MHPCC’s  largest  machine,  how 
are  they  interacting  with  the  site?  We  found  that  regional  customers  are  far  more  likely  to  use 
smaller  HPC  assets  and  MHPCC’s  knowledge  capital.  Specifically,  we  found  that  customers 
within  the  region  are  most  likely  to  use  one  of  two  capabilities:  MHPCC’s  ability  to  stand  up, 
operate,  and  maintain  custom  HPC  assets,  and  MHPCC’s  resident  programming  expertise  in 
optimizing  algorithms  for  HPC  architectures.  To  provide  context  around  these  two  capabilities, 
we  offer  two  examples  that  we  observed  during  the  course  of  our  research. 
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Based  on  a  request  from  PACOM/CC  to  increase  the  rate  at  which  operational  plans  could  be 
developed,  PACAF  approached  MHPCC  in  March  2011  about  hosting  a  dedicated  machine  to 
support  these  calculations.  As  a  result  of  this  request,  MHPCC  stood  up  an  80-core  Dell 
PowerEdge  machine  named  Anamake,  and  the  computation  time  for  a  single  run  was  reduced 
from  12  hours  to  less  than  10  minutes.  This  significant  reduction  in  time  allows  weaponeers  to 
develop  high-fidelity  solutions  by  running  more  iterations  in  shorter  amounts  of  time. 

The  technical  experts  that  run  the  modeling  software  come  from  three  groups:  cruise  missile 
support  activity  (CMS A)  Pacific  (PAC);  613  Air  Operations  Center  (AOC)  Intelligence, 
Surveillance,  and  Reconnaissance  Division  (ISRD);  and  607  AOC  ISRD. 

According  to  our  interviews,  there  are  several  benefits  to  having  this  capability  at  MHPCC. 
First,  one  staff  member  anecdotally  told  us  that  it  is  difficult  to  find  suitable  space  for  computing 
resources  on  Oahu,  noting  that  finding  the  actual  real  estate  in  a  server  room  and  setting  up  the 
information  technology  infrastructure  are  often  the  biggest  road  blocks  when  establishing  a  new 
computing  resource.40  MHPCC  had  both  the  floor  space  along  with  established  infrastructure,  so 
placing  the  machine  on  Maui  was  relatively  straightforward. 

Customers  also  noted  that  having  an  HPC  within  the  PACOM  geographic  region  means  that 
MHPCC  personnel  understand  the  command’s  mission  space  and  culture.  Several  PACOM 
staffers  told  us  that  having  an  HPC  in  theater  resulted  in  more  responsive  service,  something  they 
had  not  experienced  when  using  other  HPCMP  sites.41  As  one  person  told  us:  “Having  MHPCC 
in  theater  means  we  do  not  have  to  worry  about  the  ‘out  of  sight,  out  of  mind’  problem  that  we 
sometimes  see  when  working  with  other  resources  on  the  mainland.”42 

One  potential  issue  that  we  observed  for  this  use  case  relates  to  the  mismatch  between 
PACOM’s  operational  needs  and  MHPCC  being  primarily  a  research-and-development  resource. 
PACAF/A2  representatives  noted  they  are  taking  a  risk  in  using  MHPCC  to  host  the 
weaponeering  software  because  MHPCC  is  not  a  24/7  operation.  However,  in  the  same  sentence, 
they  noted  they  are  willing  to  take  that  risk  because  the  hosted  server  significantly  reduces  the 
processing  time.  As  one  weaponeer  told  us:  “We  are  not  executing  [operational]  plans  right  now. 
If  we  ever  have  to  start  doing  that,  the  existing  resources  may  be  insufficient.”43 


37 

MHPCC,  “Integrated  Munitions  Effects  Assessment  Technical  Interchange  Meeting,”  trip  report,  signed  by 
Thomas  Gemuend,  Air  Force  Research  Laboratory,  April  9,  2014. 

38  PACAF/A2  personnel,  2014;  613  AOC  personnel,  2014. 

39  613  AOC  personnel,  2014;  MHPCC,  2014. 

40 

PACOM/J8  personnel,  2015. 

41  PACAF/A2  personnel,  2014;  PACOM/J8  personnel,  2014;  PACOM/J8  personnel,  2015;  613  AOC  personnel, 
2014. 

42 

PACAF/A2  personnel,  2014;  PACOM/J8  personnel,  2014. 

43  PACAF/A2  personnel,  2014;  613  AOC  personnel,  2014. 
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Since  November  2014,  the  software  team  at  MHPCC  has  been  supporting  the  Range  Safety 
Office  (RSO)  at  PMRF  in  Kauai.  When  it  participates  in  a  missile  test,  PMRF  may  launch  one  or 
more  of  the  test  assets  from  Kauai,  or  it  may  simply  observe  a  test  that  originates  from  another 
location.  In  either  case,  before  every  test,  safety  officers  run  Monte  Carlo  simulations  that  predict 
where  debris  is  likely  to  fall  under  nominal  and  abnormal  conditions.  The  results  of  these 
calculations  are  used  by  the  Federal  Aviation  Administration  to  plan  “no-fly  zones”  during  the 
test,  and  the  Coast  Guard  issues  similar  warnings  for  marine  surface  vessels.44 

The  safety  officers  at  PMRF  initially  approached  MHPCC  personnel  because  they  were 
suffering  from  the  same  problem  as  the  PACOM  weaponeers — their  Monte  Carlo  runs  were 
taking  too  long  to  implement  on  their  existing  hardware  45  Like  all  Monte  Carlo  simulations,  a 
large  number  of  analytic  runs  are  needed  in  order  to  generate  a  meaningful  result,  and  the  safety 
officers  had  to  start  their  calculations  six  weeks  ahead  of  the  launch  date  in  order  to  ensure  they 
would  be  completed  in  time.46  Of  course,  if  any  key  element  of  the  planned  test  procedure  was 
changed  during  this  time,  the  RSO  would  have  to  begin  the  calculation  anew,  potentially 
delaying  the  start  of  the  test. 

In  addition  to  the  need  for  faster  computing  cycles,  PMRF’s  Monte  Carlo  runs  also  need  a  lot 
of  networking  bandwidth  and  a  classified  computing  environment.  Each  test  configuration 
requires  data  to  be  transmitted  to  the  computer  that  is  performing  the  calculation,  and  it  is 
challenging  to  transmit  this  amount  of  data  over  the  Nonsecure  Internet  Protocol  Router  Network 
(NIPRNet)  or  Secret  Internet  Protocol  Router  Network  (SIPRNet).47  The  final  requirement  is  that 

AO 

these  calculations  run  on  a  classified  SECRET  machine,  because  of  the  sensitivity  of  the  data. 

Because  of  some  restrictions  associated  with  PMRF  contracting  guidelines,  PMRF  personnel 
decided  to  turn  to  the  HPC  program  rather  than  stand  up  their  own  machine.49  To  solve  this 
problem,  MHPCC  engineers  worked  with  PMRF  personnel  to  develop  a  solution  that  runs  on 
Army  Research  Lab’s  (ARL’s)  Hercules  computer  at  Aberdeen  Proving  Ground  in  Maryland.50 
(The  analysis  is  run  in  Maryland  because  MHPCC  does  not  have  a  suitable  SECRET-level 
machine  for  this  purpose.)  Initially,  MHPCC  personnel  logged  into  the  ARL  machine  on 
PMRF’s  behalf  because  ARL  was  short-staffed  and  unable  to  support  PMRF  directly.  In  fact, 
this  problem  was  eventually  solved  when  a  system  administrator  from  MHPCC  traveled  to 
Aberdeen  for  a  week  in  order  to  automate  the  workflow.51 

44 

Pacific  Missile  Test  Range  range  safety  personnel,  2015. 

45 

Pacific  Missile  Test  Range  personnel,  2014. 

46 

Pacific  Missile  Test  Range  personnel,  2014;  Pacific  Missile  Test  Range  range  safety  personnel,  2015. 

47 

Pacific  Missile  Test  Range  range  safety  personnel,  2015. 

48 

Pacific  Missile  Test  Range  personnel,  2014. 

49 

Pacific  Missile  Test  Range  personnel,  2014. 

50  AFRL/RDSM  personnel,  2015c. 

51  AFRL/RDSM  personnel,  2015c. 
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PMRF  personnel  reported  that  MHPCC’s  onsite  expertise  was  instrumental  in  facilitating  the 
relationship  with  ARL’s  machine  and  then  providing  onsite  support  to  get  the  analysis  up  and 
running.  Specifically,  MHPCC’s  knowledge  of  both  the  local  client  and  the  HPC  enterprise  was 
what  made  it  successful  in  this  instance.  As  a  result  of  MHPCC  standing  up  these  services, 
PMRF  has  reduced  the  time  of  running  a  single  iteration  from  84  hours  down  to  less  than  five 
minutes.  Like  the  weaponeering  mission,  this  increase  in  efficiency  allows  PMRF  safety 
officers  to  develop  a  higher- fidelity  result  in  a  much-shorter  time. 

Anecdotes  from  PACOM  Highlight  Value  of  MHPCC 

As  we  mentioned  earlier,  none  of  the  stakeholders  that  we  met  with  were  able  to  provide 
quantitative  requirements  for  how  supercomputing  resources  are  needed  to  meet  their  mission, 
nor  were  they  able  to  demonstrate  a  clear  strategy  for  generating  these  requirements  in  the  near- 
term  future.  However,  while  we  were  not  able  to  find  a  clear  requirement  for  supercomputing 
resources,  we  did  hear  several  anecdotes  for  how  MHPCC  adds  value  to  PACOM.  We  recognize 
these  are  only  stories,  but  we  present  them  here  because  we  heard  them  from  multiple  parties, 
which  suggest  they  are  salient. 

The  most-common  theme  that  we  heard  is  related  to  the  proximity  effect.  Commanders  and 
members  of  the  Senior  Executive  Service  (SES)  told  us — either  directly  or  using  language  that 
implied  as  much — that  they  liked  having  an  HPC  asset  within  the  AOR.  Several  seniors  and 
staffers  lamented  that  PACOM  is  geographically  isolated,  and  having  resources  near  Oahu  is 
always  the  preferred  solution  because  this  is  often  the  most-responsive  option.  Or,  as  one  staffer 
told  us,  commanders  like  having  resources  “within  sight  of  the  flagship.” 

This  effect  is  likely  happening  because  of  cultural  and  organizational  factors.  First,  having  an 
HPC  resource  within  the  AOR  means  that  some  element  of  it  is  likely  to  fall  under  PACOM ’s 
chain  of  command,  and  this  is  useful  when  pursuing  operational  missions.  In  the  case  of 
MHPCC,  PACAF/A2  helps  maintain  MHPCC’s  network  connectivity  to  the  rest  of  the  Hawaiian 
islands,  so  both  of  these  parties  are  motivated  to  remain  responsive  to  one  another’s  needs.54 

Several  of  the  stakeholders  that  we  spoke  with  also  liked  that  MHPCC  personnel  worked 
within  the  same  time  zone,  which  makes  it  easier  for  PACOM  personnel  to  coordinate  technical 
exchanges  and  receive  support  from  MHPCC.  However,  it  is  worth  recognizing  that  it  would  be 
straightforward  to  stand  up  a  PACOM  support  cell  on  the  mainland  that  is  open  during  business 
hours  in  Hawaiian  standard  time. 


52 

Pacific  Missile  Test  Range  range  safety  personnel,  2015. 

53  PACOM/J8  personnel,  2015. 

54  PACAF/A2  personnel,  2014. 
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AMOS  and  MHPCC  Benefit  from  Co-Location 


MHPCC’s  longest  relationship  is  the  one  with  the  co-located  Air  Force  Maui  Optical  and 
Supercomputing  Site  (AMOS).  AMOS  is  operated  by  the  AFRL  and  consists  of  three  large 
(meter-class)  optical  telescopes  that  provide  tracking,  detection,  and  identification  of  orbiting 
objects.  Together,  these  telescopes  provide  the  Air  Force  with  a  unique  imaging  capability  over 
the  Pacific  that  supports  key  Air  Force  Space  Command  requirements  for  space  situational 
awareness. 

From  the  early  days  of  MHPCC,  AMOS  scientists  have  taken  advantage  of  the  sites’  co- 
location  by  leveraging  the  resident  computing  infrastructure  and  expertise  to  help  advance 
AMOS  core  capabilities.  Today,  AFRL’s  Remote  Maui  Experiment  facility  provides  floor  space 
for  an  864-core  server  that  is  dedicated  to  processing  the  images  from  the  observatory’s  1.6- 
meter  telescope.55  This  server  runs  an  AFRL-developed  algorithm  that  compensates  for 
atmospheric  blur  in  order  to  generate  sharp  images  of  overhead  satellites  in  near-real  time.  These 
resulting  images  are  then  sent  to  operational  users. 

Today,  the  relationship  between  MHPCC  and  AMOS  is  quite  integrated,  especially  after  a 
2014  move  that  co-located  the  MHPCC  and  AMOS  administrative  offices  to  Building  50. 56  This 
co-location  fosters  a  significant  sharing  of  expertise  between  the  two  sites,  but  it  also  results  in  a 
cost  savings  for  both  sites  because  they  share  some  of  the  expenses  associated  with  the  building 
lease,  maintenance  costs,  and  support  infrastructure.  In  fact,  MHPCC  personnel  reported  to  us 
that  DoD  gains  an  efficiency  of  about  $2.9  million  a  year  by  having  AMOS  and  MHPCC  co¬ 
located.57 

MHPCC  also  benefits  from  being  co-located  with  the  observatory.  AFRL  is  a  research- 
oriented  organization,  and  MHPCC  has  benefited  by  being  surrounded  by  this  culture.  For 
example,  the  HPCMP's  Portal  program — a  capability  that  HPCMP  notes  as  an  innovative  and 
significant  contribution  to  its  computing  enterprise — was  initiated  and  is  still  led  by  the 
personnel  who  developed  the  control  system  for  the  optical  observatory.  The  combined 
presence  of  AMOS  and  MHPCC  personnel  provides  a  critical  mass  for  capturing  and 
maintaining  world-class  research  talent. 


55  MHPCC  DSRC,  2014b. 

56  AFRL/RD  personnel,  phone  conversation  with  author,  June  3,  2014. 

57  AFRL/RDSM  personnel,  2015b. 

CO 

DoD  HPC  Modernization  Office  personnel,  2014;  AFRL/RDSM  personnel,  2015c. 
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4.  Recommendations 


We  began  this  report  by  listing  a  set  of  three  research  questions:  Which  of  MHPCC’s  capabilities 
are  used  most  often  by  customers  in  the  Pacific  AOR?  And,  specifically,  is  there  a  demand  for  a 
supercomputing  resource  that  is  located  within  the  AOR?  What  recommendations  should  AFRL 
consider  when  making  decisions  about  the  future  of  MHPCC? 

The  findings  that  we  describe  in  the  preceding  chapters  addressed  the  first  and  second 
questions.  This  chapter  offers  a  set  of  six  recommendations  that  AFRL  might  consider  when 
strategizing  about  the  future  of  the  site.  These  are  not  exhaustive  recommendations. 

Recommendation  1:  Be  explicit  to  Pacific  AOR  users  about  the  different  capabilities  at 
MHPCC 

Throughout  our  interviews,  we  observed  a  general  lack  of  understanding  among  the  Pacific 
AOR  customer  base  about  what  capabilities  are  available  at  MHPCC,  which  reside  in  the  greater 
HPCMP  enterprise,  and  which  belong  to  another  government  or  commercial  party. 

In  its  basic  form,  this  misunderstanding  manifests  itself  as  a  lack  of  clarity  about  which 
machine  is  used  to  solve  a  specific  problem.  For  example,  we  found  one  user  who  mistakenly 
believed  that  her  job  was  running  on  a  machine  in  Maui  when,  in  reality,  it  ran  on  an  HPC  asset 
on  the  U.S.  mainland.  This  misconception  was  not  due  to  any  false  advertising  on  the  part  of 
MHPCC,  but  it  does  highlight  the  challenge  of  managing  a  virtual  workflow,  in  which  the  user 
cannot  directly  observe  the  hardware. 

One  way  to  address  this  problem  would  be  to  develop  a  simple  architecture  diagram  that 
shows  both  the  entire  HPCMP  enterprise  along  with  the  details  on  MHPCC-specific  resources. 
Having  this  diagram  would  allow  MHPCC  personnel  to  first  explain  how  DoD’s  HPC  enterprise 
works,  before  focusing  on  the  resources  that  are  local  to  Maui.  Displaying  such  a  diagram  would 
also  highlight  the  importance  of  the  DREN  network,  which  is  what  stitches  together  the  entire 
architecture. 

In  the  case  of  this  example,  walking  the  customer  through  such  a  diagram  will  help  show 
how  having  MHPCC’s  presence  in  Hawaii  helps  connect  users  to  the  rest  of  the  system  on  the 
U.S.  mainland. 

Recommendation  2:  MHPCC  should  be  described  to  Pacific  AOR  customers  as  offering 
three  separate  products 

Every  customer  is  different,  and  the  answer  to  a  client’s  computing  problem  needs  to  be  more 
nuanced  than  “MHPCC.”  To  help  address  this  problem,  we  propose  that  MHPCC  market  itself  as 
offering  three  separate  products:  (1)  a  large,  12,096-node  supercomputer;  (2)  a  facility  that  has 


21 


the  expertise  and  infrastructure  to  host  standalone  assets  for  local  clients;  and  (3)  a  research  staff 
that  has  the  capacity  to  develop  innovative  solutions  to  operational  problems. 

Presenting  MHPCC  as  a  center  that  offers  separate  products  will  allow  for  a  more  tailored 
approach  to  solving  Pacific  AOR  customer  problems.  It  would  also  help  address  the  perception 
that  MHPCC  is  only  the  Riptide  machine.  For  example,  if  Riptide  is  not  appropriate  for  a 
specific  customer's  problem,  then  it  is  easy  to  set  that  asset  aside  and  start  discussing  the  part  of 
MHPCC  that  offers  development  and  hosting  of  tailored,  standalone  solutions.  By  separating 
these  capabilities,  MHPCC  and  HPCMP  can  target  development  resources  to  the  parts  of  the 
product  line  that  will  ultimately  provide  the  most  utility  to  stakeholders  in  the  AOR. 

Recommendation  3:  Provide  the  Pacific  AOR  customer  with  a  framework  for  matching 
HPC  assets  to  the  desired  end  result 

Another  challenge  associated  with  HPC — besides  the  fact  that  it  is  often  utilized  virtually  — 
is  that  it  is  inherently  technical.  It’s  hard  to  engage  and  strategize  with  customers  when  the 
product  is  virtual  and  requires  specialized  knowledge  to  use  effectively.  What  is  lacking  right 
now  in  MHPCC ’s  customer  engagement  is  a  way  to  work  through  these  issues  alongside  the 
customer  in  the  AOR,  so  both  parties  understand  the  problem  and  how  HPC  will  be  used  to 
address  it. 

We  recommend  implementing  a  simple  framework  for  helping  Pacific-based  clients 
understand  the  capabilities  and  differences  between  HPC  assets.  Table  1.1  represented  one  such 
approach  for  distinguishing  between  cluster  and  supercomputing  architectures.  The  information 
in  Table  1.1  can  be  taken  a  step  further  and  turned  into  a  flowchart  that  allows  the  customer  to 
determine  what  kind  of  HPC  asset  is  most  suitable  for  his  or  her  problem,  Figure  4.1. 

The  flowchart  that  we  show  in  Figure  4.1  is  a  simplistic  example,  but  it  offers  a  way  for  the 
customer  to  put  a  problem  in  context  with  the  available  resources.  Such  a  flowchart  is  one  way  to 
counter  the  perception  that  a  supercomputer  is  “anything  larger  than  a  desktop.”  It  is  also  a 
simple  way  to  start  educating  Pacific  AOR  users  on  the  available  types  of  HPC  assets;  this  can 
serve  as  a  first  step  for  customers  to  begin  incorporating  HPC  capabilities  into  their  strategic 
plans. 

One  could  also  imagine  analogous  flowcharts  where  the  possible  end  states  list  all  of 
MHPCC ’s  current  capabilities  so  customers  can  see  where  their  problem  lands — if  at  all — within 
MHPCC ’s  enterprise.  Telling  customers  that  they  would  be  better  served  by  renting  space  on  a 
commercial  system — or  buying  their  own  standalone  machines — still  solves  their  problem,  and  it 
frames  MHPCC  personnel  as  being  a  knowledgeable  partner  who  has  customers’  best  interest  in 
mind. 
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Figure  4.1.  An  Example  Flowchart  for  Matching  Workload  to  Computing  Assets 


SOURCE:  Authors’  analysis. 
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Recommendation  4:  Employ  both  strategic  and  technical  thought  leaders  to  enhance 
Pacific  AOR  customer  exploitation  of  MHPCC  capabilities 

Of  course,  developing  a  clever  flowchart  or  table  is  only  part  of  the  solution.  The  other  half 
requires  a  two-part  customer  engagement  strategy:  one  part  HPC  subject-matter  expert  and  one 
part  strategist.  (We  specify  two  parts  because  these  skills  are  likely  to  reside  in  two  separate 
individuals  or  teams  of  individuals.)  The  strategists  need  to  identify  areas  within  PACOM  that 
might  benefit  from  HPC  capabilities,  and  subject-matter  experts  need  to  work  with  PACOM 
personnel  to  help  them  develop  requirements  for  HPC  assets  that  will  solve  their  problem. 

For  example,  PACOM/J9  has  some  bold  visions  for  future  missions,  but  they  are  not  HPC 
experts.  J9  will  require  objective  HPC  subject-matter  expertise  to  help  them  identify  what  types 
of  new  computing  resources  will  be  needed  to  help  them  realize  their  goals.  The  first  step  in  this 
relationship  might  be  an  accessible  seminar  from  MHPCC  personnel  on  the  differences  between 
modem  computing  architectures.  This  might  graduate  to  conversations  where  MHPCC  personnel 
help  J9  begin  developing  requirements.  The  ultimate  goal  of  this  progression  should  be  to  give  J9 
personnel  enough  context  on  HPC  resources  that  they  begin  to  think  of  use  cases  and  first-order 
requirements  on  their  own. 

Recommendation  5:  Rebuild  trust  and  understanding  with  HPCMP  leadership 

MHPCC  needs  to  communicate  more  clearly  and  more  frequently  with  HPCMP  management 
to  rebuild  the  trust  that  was  lost  as  a  result  of  the  aforementioned  contracting  issues. 
Reestablishing  trust  is  always  a  challenging  task,  but  MHPCC  personnel  can  take  some  comfort 
in  the  fact  that  the  HPCMP  still  recognizes  Maui  as  being  a  source  of  innovation  and  creativity 
for  the  entire  enterprise.59 

The  three  contracting  and  program  management  challenges  that  we  outlined  earlier  are 
important  to  acknowledge  as  mistakes.  Like  all  missteps,  it  is  important  to  learn  from  them  and 
then  begin  thinking  about  the  future,  rather  than  dwelling  on  the  past.  As  MHPCC  begins  to 
repair  the  relationship,  focusing  on  simple  things  like  establishing  consistent  communication  to 
provide  updates  is  a  good  first  start.  As  we  have  observed  with  other  Hawaii-based  programs  that 
are  managed  from  the  U.S.  mainland,  it  is  very  easy  for  the  sponsoring  organization  to  lose 
context  on  what  progress  is  being  made  if  the  communication  between  the  two  parties  is  irregular 
and  inconsistent. 

Recommendation  6:  Recognize  that  aspects  of  the  computing  industry  are  changing. 
Develop  a  strategy  to  prepare  for  this  change. 

Throughout  the  course  of  performing  this  research,  we  observed  two  trends  that  are  worth 
considering  when  developing  a  strategy  for  MHPCC’s  future. 


59  DoD  HPC  Modernization  Office  personnel,  2014. 
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First,  we  saw  evidence  to  suggest  that  the  supercomputing  industry  is  changing.  New, 
scalable  computing  architectures,  combined  with  a  drop  in  the  price  of  hardware,  have  the 
potential  to  turn  computing  cycles  into  a  commodity.60  The  supply  side  of  computing  is  changing 
to  reflect  the  shift:  The  major  supercomputing  developers — IBM  and  Cray — have  expanded  their 
product  lines  to  offer  analytics  hardware,  in  addition  to  their  conventional,  supercomputing 
offerings.  As  a  result,  a  larger  portion  of  today’s  research  and  development  dollars  are  going 
toward  developing  machines  that  will  be  better  suited  for  addressing  big  data  applications.  With 
all  of  these  changes,  HPC  leaders  should  be  addressing  the  following  question:  How  will  centers 
provide  value  when  the  world  has  access  to  inexpensive  computing  cycles? 

The  second  trend  that  we  observed  offers  one  potential  solution  to  this  question.  In  the  course 
of  our  research,  we  noted  that  MHPCC’s  desire  to  perform  work  for  others  mirrors  what  we 
observed  within  the  Department  of  Energy’s  (DoE)  HPC  program.  DoE,  of  course,  has  its  own 
supercomputing  facilities,  and  it  reported  that  it  has  started  to  become  more  aggressive  in 
developing  partnerships  with  end  users  and  applications  researchers.61  DoE  recognizes  that 
simply  having  the  world’s  fastest  machines  is  not  going  to  remain  its  primary  competitive 
advantage  for  more  than  one  or  two  additional  computing  generations.  Instead,  DoE’s  value 
will  come  from  having  the  foresight  to  develop  expertise  in  today’s  emerging  problems  and 
figuring  out  how  HPC  resources  can  be  used  to  address  those  problems. 

MHPCC  is  in  the  early  stages  of  taking  up  this  posture,  but  adopting  this  as  a  formal  strategy 
is  going  to  require  some  strategic  and  organizational  adjustments.  As  an  example,  HPCMP  is 
funded  as  an  RDT&E  enterprise,  but  many  of  the  new  customers  that  MHPCC  is  pursuing  are 
operational  in  nature.  Reconciling  this  misalignment  offers  an  opportunity  for  MHPCC — perhaps 
even  the  entire  HPCMP — to  offer  support  to  a  greater  number  of  government  customers,  just  as 
the  demand  for  and  awareness  of  these  powerful  analytic  tools  is  likely  to  increase  significantly, 
while  maintaining  the  critical  support  to  the  RDT&E  community. 

Combined  with  its  unique  geographical  placement,  MHPCC’s  increasing  expertise  in 
responding  to  support  requests  from  the  operational  community  will  provide  unique  value  to 
HPCMP  and  DoD  as  a  whole.  For  example,  MHPCC  may  be  able  to  provide  significant  latency 
arbitrage  services  for  future  Pacific  AOR  real-time  jobs.  It  can  serve  in  a  backup  capacity  for 
low-priority  and  high-latency  research  jobs  from  HPCMP ’s  affiliates  in  the  continental  United 
States.  MHPCC  can  also  serve  as  a  test  bed  for  developing  best  practices  for  managing 


60  “The  Cheap,  Convenient  Cloud,”  The  Economist,  April  18,  2015;  Gartner,  “Gartner  Says  Cloud  Application 
Infrastructure  Technologies  Need  Seven  Years  to  Mature,”  press  release,  February  2,  2009;  Radu  Sion,  “To  Cloud 
or  Not  to?  Musings  on  Clouds,  Security  and  Big  Data,”  in  Secure  Data  Management,  Vol.  8425,  May  2014,  pp.  3-5; 
Yao  Chen  and  Radu  Sion,  “To  Cloud  or  Not  to  Cloud?  Musings  on  Costs  and  Viability,”  in  Proceedings  of  the  2nd 
ACM  Symposium  on  Cloud  Computing,  ACM,  2011,  p.  29. 

61  Brase,  2015. 

62  Brase,  2014. 
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operational  and  RDT&E  workloads  within  the  same  service.  HPCMP  focuses  on  RDT&E  and 
may  benefit  from  MHPCC’s  experience. 
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5.  Conclusion 


In  conclusion,  we  found  that  MHPCC  has  capabilities  and  expertise  that  provide  value  to 
regional  users.  While  we  did  not  find  anyone  within  the  Pacific  AOR  who  currently  requires  the 
use  of  the  Riptide  machine  to  support  operations,  there  are  several  users  who  rely  on  MHPCC ’s 
ability  to  stand  up  and  support  custom  hardware.  MHPCC ’s  software  development  capability  is 
also  recognized  by  both  the  customer  base  and  the  HPCMP  as  being  exemplary. 

In  the  near  term,  MHPCC  should  take  steps  to  educate  the  Pacific  customer  base  and  build 
trust  with  the  HPCMP.  For  the  customer  base,  all  of  the  site’s  potential  users  will  need  to  be 
better  informed  as  to  the  site’s  capabilities  and  how  these  can  be  integrated  into  a  user’s  mission. 
The  technical  staffers  at  MHPCC  will  play  a  significant  role  in  this  because  of  their  experience  in 
software  development.  They  also  represent  a  capacity  for  developing  the  next  generation  of 
computing  technology,  like  the  quantum  computing  work  that  is  being  done  at  MHPCC  and  the 
University  of  Hawaii. 

As  AFRL,  MHPCC,  the  Army,  and  Congress  work  through  the  tradeoffs  of  balancing 
resources  by  eliminating  unnecessary  redundancies  and  costs,  it  is  important  that  they  retain  the 
capabilities  on  which  DoD  users  depend.  The  site  provides  value  to  a  key  AOR  in  supporting 
two  important  tasks  within  the  geographic  region.  In  addition,  there  is  a  strong  likelihood  that 
AOR  users  will  leverage  MHPCC  capabilities  in  the  future.  As  a  result,  users  in  the  AOR  would 
likely  need  to  find  alternative  solutions  if  the  MHPCC  is  cut  entirely. 
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